Curse Of Dimensionality Research Articles

Machine Learning and Feature Selection for Soil Spectroscopy. An Evaluation of Random Forest Wrappers to Predict Soil Organic Matter, Clay, and Carbonates

Soil spectroscopy estimates soil properties using the absorption features in soil spectra. However, modelling soil properties with soil spectroscopy is challenging due to the high dimensionality of spectral data. Feature Selection wrapper methods are promising approaches to reducing dimensionality but are barely used in soil spectroscopy. The aim of this study is to evaluate the performance of two feature selection wrapper methods, Sequential Forward Selection (SFS) and Sequential Flotant Forward Selection (SFFS) built using the Random Forest (RF) algorithm, for dimensionality reduction of spectral data and predictive modelling of modelling soil organic matter (SOM), clay and carbonates. The reflectance of 100 soil samples, acquired from Sierra de las Nieves (Spain), was measured under laboratory conditions using ASD FieldSpec Pro JR. Four different datasets were obtained after applying two spectral preprocessing methods to raw spectra: raw spectra, Continuum Removal (CR), Multiplicative Scatter Correction (MSC), and a so-called “Global” dataset composed of raw, CR and MSC features. The performance of RF models built with feature selection methods was compared to that of Partial Least Squares Regression (PLSR) and RF (alone).RF models built with SFS and SFFS outperformed PLSR and RF alone models: The best RF models with feature selection had a respective ratio of performance to interquartile distance of 1.93, 0.38 and 2.56. PLSR models had an accuracy of 1.41, 0.29 and 1.81 for SOM, carbonates, and clay, respectively. RF alone had a respective performance of 1.29, 0.29 and 1.81. The application of feature selection wrapper methods reduced the number of features to less than 1% of the starting features. Features were selected across all spectra for SOM and clay, and around 900 nm, 1900 nm, and 2350 nm for carbonates. However, feature selection highlighted features around 1100 nm in SOM modelling, as well as other features around 2200 nm, which is considered a main absorption feature of clay. The application of feature selection with Random Forest was very important in improving modelling accuracy, reducing the redundant features and avoiding the curse of dimensionality or Hughes effect. Thus, this research showed an alternative to dimensionality reduction approaches that have been applied to date to model soil properties with spectroscopy and paves the way for further scientific investigation based on feature selection methods and machine learning.

Handling noise and overfitting in surrogate models based on non-uniform rational basis spline entities

This paper proposes a general and systematic approach to generate a surrogate model based on non-uniform rational basis spline (NURBS) hyper-surfaces as a solution of a constrained non-linear programming problem, which is solved through a gradient-based algorithm by considering two formulations. The first formulation, which is the general one, is prone to the curse of dimensionality. The second one separates dimensions in the NURBS domain and allows saving computational time and resources. The optimisation process consists of three steps, which are solved sequentially. Each step is characterised by different design variables and solving strategies. The gradient of the cost function, for both formulations and for each optimisation step, is derived analytically by exploiting the properties of NURBS entities, hence enhancing computational efficiency and accuracy. One of the main features of the presented methodology lies the incorporation of a smoothing term in the problem formulation. This term plays a crucial role in controlling overfitting phenomenon and attenuating local oscillations within the dataset. These oscillations may be due to various sources, such as inherent noise, model instabilities, measurement uncertainties, or numerical precision limitations. The effectiveness of the proposed approach is tested on both analytical benchmark problems and real-world engineering problems taken from the literature.

A Sugeno ANFIS Model Based on Fuzzy Factor Analysis for IS/IT Project Portfolio Risk Prediction

Risk inherence jeopardises Information System (IS) and Information Technology (IT) Project Portfolio Management (PPM) to realise thestrategic objectives. Previous studies have mainly provided Artificial Intelligence (AI) and statistical models to predict the overall riskof IS/IT project portfolio, whereas neuro-fuzzy models were rarely used. This paper proposes a Sugeno Adaptive Neuro-Fuzzy InferenceSystem (ANFIS) model based on Fuzzy Factor Analysis (FFA) named ANFIS-OPR to predict the overall risk of IS/IT project portfolio fromhistorical IS/IT project risk data. The ANFIS-OPR inputs are the relevant factor loadings resulting from the FFA application on the IS/IT projects risks set to cope with the curse of dimensionality. Then, the Sugeno ANFIS model is adopted to give strategic interpretability to the predicted IS/IT project portfolio overall risk by implementing the IS/IT Project Management Office (PMO) expert knowledge, represented by fuzzy rules, on the relationship between IS/IT project portfolio strategic alignment and the IS/IT projects risks. The ANFISOPR outputs are the predicted Overall Portfolio Risk (OPR) and Root Mean Square Error (RMSE). The paper also presents an IS/IT PMO case study that shows the proposed ANFIS-OPR efficacy, which predicted the OPR values closely to the OPR estimates with an accepted RMSE of 0.108. The proposed ANFIS-OPR is a novel intelligent decision-making tool that enables the IS/IT PMO to monitor the OPR, considering its linkage with strategic alignment; thus, contingency plans can be carried out appropriately while ensuring that the IS/IT project portfolio is strategically aligned.

From Text to Recommendations: How Vector Databases are Revolutionizing Personalized Content Delivery

Abstract: Effective similarity search and retrieval are now possible thanks to vector databases, which have become a potent tool for organizing and searching high-dimensional data. This article provides a thorough examination of vector databases, their underlying theories, and their applications across a range of industries. In handling complex data types, we address the significance of vector representation and emphasize the benefits of vector databases over traditional databases [1]. The article explores the process of creating a vector database, highlighting the critical function of indexing strategies such as IVF (Inverted File Indexing) and HNSW (Hierarchical Navigable Small World) in guaranteeing the effectiveness and precision of searches [2]. In addition, we discuss the problems caused by the curse of dimensionality and offer solutions to lessen its effects on nearest neighbor searches [3]

Hermite kernel surrogates for the value function of high-dimensional nonlinear optimal control problems

Numerical methods for the optimal feedback control of high-dimensional dynamical systems typically suffer from the curse of dimensionality. In the current presentation, we devise a mesh-free data-based approximation method for the value function of optimal control problems, which partially mitigates the dimensionality problem. The method is based on a greedy Hermite kernel interpolation scheme and incorporates context knowledge by its structure. Especially, the value function surrogate is elegantly enforced to be 0 in the target state, non-negative and constructed as a correction of a linearized model. The algorithm allows formulation in a matrix-free way which ensures efficient offline and online evaluation of the surrogate, circumventing the large-matrix problem for multivariate Hermite interpolation. Additionally, an incremental Cholesky factorization is utilized in the offline generation of the surrogate. For finite time horizons, both convergence of the surrogate to the value function and for the surrogate vs. the optimal controlled dynamical system are proven. Experiments support the effectiveness of the scheme, using among others a new academic model with an explicitly given value function. It may also be useful for the community to validate other optimal control approaches.

Integrative Analysis of Site-Specific Parameters with Nuisance Parameters on the Common Support

AbstractHigh-throughput technologies in bioscience have pushed us into an era with high dimensionality. Swamped by thousands of predictors, detecting the valuable signal from the noise in clinical studies becomes challenging. As a common strategy, integrative analysis utilizing similarities across multiple studies might help lift the curse of dimensionality and enhance statistical power. However, due to the growing concern about individual data privacy, data-sharing constraints are often imposed in integrative analysis. These might lead to results inequivalent to ones without sharing constraints and reduce statistical power in integrative analyses. In this paper, built on Abess, we propose an integrative analysis method to estimate the site-specific parameters in the presence of high dimensional nuisance parameters in multi-site studies. Implemented with a carefully designed $$L_{2,0}$$ L 2 , 0 penalization on nuisance parameters, the proposed method satisfies both the DataSHIELD constraint, which only allows the transmission of summary statistics from sites, and the equivalence property that the solution is exactly the same as the solution merging all datasets into one on a single location. Assuming the nuisance parameters share a common support, the proposed method has support recovery and selection consistency with high probability and exhibits improved estimation accuracy on the site-specific parameters and low computational cost in numerical experiments. We demonstrate the merit of the proposed method by investigating the relationship between the CD8 T cell count and the treatment effect of zidovudine-incorporated therapy in the AIDS Clinical Trials Group Study 175.

Transportation infrastructure upgrading and green development efficiency: Empirical analysis with double machine learning method

In order to deal with the environmental problems such as pollution emissions and climate change, sustainable development in the field of transportation has gradually become a hot topic to all sectors of society. In addition, promoting the green and low-carbon transformation of China's transportation is also an important issue in the new era. Thus, it is particularly important to correctly identify the green effect of high-speed rail. However, the traditional causal reasoning model faces several challenges such as ‘dimensional curse’ and multicollinearity. Based on the panel data of 283 prefecture-level cities in China from 2003 to 2019, this study uses the double machine learning model to explore the impact of transportation infrastructure upgrading on the efficiency of urban green development in China. The research shows that the upgrading of transportation infrastructure can effectively improve the efficiency of urban green development by 4%. Service industry agglomeration and green innovation are verified as two mediating channels. Moreover, the synthetic difference in difference model is employed to evaluate the regional impact of high-speed rail, and finds that the regional impact of transportation policies often exceeds the impact of individual cities. We further apply the conclusions of this paper to the research at the micro enterprise level. Goodman-Bacon decomposition and a variety of robustness tests confirm the validity of our conclusions. The study's comprehensive empirical analysis not only validates the positive effects of transportation upgrades on green development, but also offers novel insights into the underlying mechanisms and policy implications of transportation upgrading.

Kernel density estimation of allele frequency including undetected alleles.

Whereas undetected species contribute to estimation of species diversity, undetected alleles have not been used to estimated genetic diversity. Although random sampling guarantees unbiased estimation of allele frequency and genetic diversity measures, using undetected alleles may provide biased but more precise estimators useful for conservation. We newly devised kernel density estimation (KDE) for allele frequency including undetected alleles and tested it in estimation of allele frequency and nucleotide diversity using population generated by coalescent simulation as well as well as real population data. Contrary to expectations, nucleotide diversity estimated by KDE had worse bias and accuracy. Allele frequency estimated by KDE was also worse except when the sample size was small. These might be due to finity of population and/or the curse of dimensionality. In conclusion, KDE of allele frequency does not contribute to genetic diversity estimation.

Augmented projection Wasserstein distances: Multi-dimensional projection with neural surface

The Wasserstein distance is a fundamental tool for comparing probability distributions and has found broad applications in various fields, including image generation using generative adversarial networks. Despite its useful properties, the performance of the Wasserstein distance decreases when data is high-dimensional, known as the curse of dimensionality. To mitigate this issue, an extension of the Wasserstein distance has been developed, such as the sliced Wasserstein distance using one-dimensional projection. However, such an extension loses information on the original data, due to the linear projection onto the one-dimensional space. In this paper, we propose novel distances named augmented projection Wasserstein distances (APWDs) to address these issues, which utilize multi-dimensional projection with a nonlinear surface by a neural network. The APWDs employ a two-step procedure; it first maps data onto a nonlinear surface by a neural network, then linearly projects the mapped data into a multidimensional space. We also give an algorithm to select a subspace for the multi-dimensional projection. The APWDs are computationally effective while preserving nonlinear information of data. We theoretically confirm that the APWDs mitigate the curse of dimensionality from data. Our experiments demonstrate the APWDs’ outstanding performance and robustness to noise, particularly in the context of nonlinear high-dimensional data.

An effective ensemble classification framework for intrusion detection system using LSTM autoencoder

In intrusion detection, the curse of dimensionality and the trade-off between maintaining a low false alarm rate and achieving a high detection rate are significant challenges. This research suggests a unique strategy based on dimensionality reduction methods to improve the performance of network intrusion detection systems (NIDS). Compressing high-dimensional network traffic data using a Long Short-Term Memory Autoencoder (LSTMAE) allows the reduced characteristics to be submitted to a classifier to identify anomalies that may indicate an attack. Using standard datasets, including Network Security Laboratory - Knowledge Discovery in Datasets (NSL-KDD), UNSW-NB15, and Canadian Institute for Cyber Security - Intrusion Detection Systems (CICIDS2017), the proposed model is tested with classifiers like Random Forest (RF) and LightGBM (Light Gradient Boosting Machines). It is hoped that by adopting this method, NIDS response times may be improved while costs associated with storing and processing data are minimized. Precision, recall, F-score, accuracy, detection rate (DR), and false alarm rate (FAR) are only a few of the performance measures used to assess the quality of the suggested models. The experimental findings show that the proposed LSTMAE model reduces prediction errors more effectively than classic machine learning techniques such as Random Forest (RF), Gradient Boosting (GB), Support Vector Machines (SVM), Deep Belief Networks (DBN), Deep Neural Networks (DNN), Autoencoder (AE), and Long Short-Term Memory (LSTM). The results also show that the proposed solution outperforms the state-of-the-art methods of detection accuracy and computing complexity using accuracy, precision, recall, F1_Score, detection rate, and FAR.

Adaptive Lagrangian Policies for a Multiwarehouse, Multistore Inventory System with Lost Sales

Adaptive Lagrangian Policies for Inventory Control Consider the problem of managing inventory in a multiwarehouse, multistore setting where each store can be periodically replenished from potentially a set of nearby warehouses. This is a practically relevant problem, and its optimal policy is not easy to compute because of the curse of dimensionality. Existing works in the literature have proposed a policy based on Lagrangian relaxation of the original stochastic problem. This vanilla policy applies the control parameters derived from the Lagrangian model and has been shown to perform well in some cases. In “Adaptive Lagrangian Policies for a Multiwarehouse, Multistore Inventory System with Lost Sales,” we go one step farther and develop adaptive Lagrangian policies that update (at certain update times) the control parameters of the vanilla policy based on the historical demand realizations. We analytically show that our proposed adjustment significantly improves the performance of the vanilla policy.

A Multi-Task Decomposition-Based Evolutionary Algorithm for Tackling High-Dimensional Bi-Objective Feature Selection

Evolutionary algorithms have been widely applied for solving multi-objective optimization problems, while the feature selection in classification can also be treated as a discrete bi-objective optimization problem if attempting to minimize both the classification error and the ratio of selected features. However, traditional multi-objective evolutionary algorithms (MOEAs) may have drawbacks for tackling large-scale feature selection, due to the curse of dimensionality in the decision space. Therefore, in this paper, we concentrated on designing an multi-task decomposition-based evolutionary algorithm (abbreviated as MTDEA), especially for handling high-dimensional bi-objective feature selection in classification. To be more specific, multiple subpopulations related to different evolutionary tasks are separately initialized and then adaptively merged into a single integrated population during the evolution. Moreover, the ideal points for these multi-task subpopulations are dynamically adjusted every generation, in order to achieve different search preferences and evolutionary directions. In the experiments, the proposed MTDEA was compared with seven state-of-the-art MOEAs on 20 high-dimensional classification datasets in terms of three performance indicators, along with using comprehensive Wilcoxon and Friedman tests. It was found that the MTDEA performed the best on most datasets, with a significantly better search ability and promising efficiency.

Augmenting input-output system under the curse of dimensionality: AHP-RAS integration and algorithm simulation

ABSTRACT In the estimation of input-output tables, one frequently encountered challenge is the aggregation of sectors in initial tables, which complicates the estimation of more detailed tables due to missing data—a phenomenon referred to as the “curse of dimensionality” when augmenting from lower to higher dimensions. We propose an AHP-RAS method to address the substantial data deficiencies encountered in the expansion of input-output tables. The method applies the Analytical Hierarchy Process (AHP) to transform the subjective relative importance of sectors into input coefficients using Pairwise Comparison Matrices (PCM) and eigenvector methods. Subsequently, it incorporates the RAS approach to supplement objective data, thereby integrating subjective judgement and objective information to tackle the curse of dimensionality. Unlike traditional AHP methods, this study employs binary integer programming to decompose large-scale matrices into subsets, making the AHP technique applicable to input-output tables with a broader range of sectors. We employ Monte Carlo simulation techniques to evaluate the robustness of the proposed methodology, specifically assessing the impacts of subjective judgement, objective data utilization, and the variation in sector numbers. Overall, the method overcomes the curse of dimensionality, yielding coefficients within an acceptable error margin, thereby demonstrating a significant advancement in the field of input-output analysis.

A hybrid nonparametric multivariate density estimator with applications to risk management

. Multivariate density estimation is plagued by the curse of dimensionality in theory and practice. We propose a hybrid density estimator of a multivariate density f that combines the strengths of the kernel estimator and the exponential series estimator. This estimator refines a preliminary kernel estimate f ̂ 0 with a multiplicative correction that estimates the ratio r = f / f ̂ 0 with an exponential series estimator. Thanks to the consistency of the pilot estimate, the coefficients of the series expansion tend to approach zero asymptotically. Accordingly, we design a thresholding method for basis function selection. A major obstacle of multivariate exponential series estimator is the calculation of its normalization factor. We resolve this difficulty with Monte Carlo integration, using the pilot kernel estimate as the trial density for importance sampling. This approach greatly enhances the practicality of the hybrid estimator. Numerical simulations demonstrate the good finite sample performance of the hybrid estimator. We present one empirical application in financial risk management.

Progressive feature reduction with varied missing data and feature selection for arthritis disease prediction

In the expansive domain of data-driven research, the curse of dimensionality poses challenges such as increased computational complexity, noise sensitivity, and the risk of overfitting models. Dimensionality reduction is vital to handle high-dimensional datasets effectively. The pilot study disease dataset (PSD) with 53 features contains patients with Rheumatoid Arthritis (RA) and Osteoarthritis (OA). Our work aims to reduce the dimension of the features in the PSD dataset, identify a suitable feature selection technique for the reduced-dimensional dataset, analyze an appropriate Machine Learning (ML) model, select significant features to predict the RA and OA disease and reveal significant features that predict the arthritis disease. The proposed study, Progressive Feature Reduction with Varied Missing Data (PFRVMD), was employed to reduce the dimension of features by using PCA loading scores in the random value imputed PSD dataset. Subsequently, notable feature selection methods, such as backward feature selection, the Boruta algorithm, the extra tree classifier, and forward feature selection, were implemented on the reduced-dimensional feature set. The significant features/biomarkers are obtained from the best feature selection technique. ML models such as the K-Nearest Neighbour Classifier (KNNC), Linear Discriminant Analysis (LDA), Logistic Regression (LR), Naïve Bayes Classifier (NBC), Random Forest Classifier (RFC) and Support Vector Classifier (SVC) are used to determine the best feature selection method. The results indicated that the Extra Tree Classifier (ETC) is the promising feature selection method for the PSD dataset because the significant features obtained from ETC depicted the highest accuracy on SVC.

Adaptive mean field multi-agent reinforcement learning

Large-scale Multi-Agent Reinforcement Learning (MARL) is fundamentally a challenge due to the curse of dimensionality. In a homogeneous multi-agent setting, mean field theory gives an effective way of scalable MARL by abstracting other agents to a virtual mean agent, assuming that the influence between agents is equal and infinitesimal. However, in some real scenarios, only several neighboring agents, rather than all agents, affect the decision-making of an agent, and different neighboring agents may have varying degrees of influence on the agent's decision-making. In this paper, not restricted to a homogeneous setting, we propose adaptive mean field MARL, which is based on the attention mechanism and can be used to deal with many-agent scenarios where there may be different influence relationships among agents. Specifically, we first derive the mean field approximation with adaptive weight and give the error bound of the approximation. Then, we propose adaptive mean field Q-Learning and describe how to obtain the adaptive weight. In addition, we discuss the differences between the proposed approach and existing mean-field MARL methods. Finally, we conduct experiments on simulation platforms, and the results show that the performance of the proposed approach outperforms that of the state-of-the-art method.

Model compression of deep neural network architectures for visual pattern recognition: Current status and future directions

Visual Pattern Recognition Networks (VPRNs) are widely used in various visual data based applications such as computer vision and edge AI. VPRNs help to enhance a machine's learning by extracting underlying patterns from input images and videos. VPRNs can be implemented using statistical, syntactical, or Machine Learning (ML) based approaches. With the popularity of Deep Neural Networks (DNNs) in the recent past, DNN based VPRNs have become an inevitable choice due to their suitability for handling such high dimensional data.However, such DNN based VPRNs bring along a curse of dimensionality, leading to intricate computations, substantial memory demands, and increased energy requirements. This impedes their practical deployment in resource constrained and strict latency required environments. Such overheads impose a demand for compression of VPRNs without impairing their performance. This research presents an exhaustive survey on compression techniques used for sequential, non-sequential and advanced DNN architectures from the perspective of Visual Pattern Recognition (VPR). Research findings of this study reveal potential scope to compress non-sequential, sequential and advanced DNN architectures for VPR applications. The study also clarifies challenges, reported and unreported issues as well as research gaps and provides directions for further research in this domain.

A novel kriging-improved high-dimensional model representation metamodelling technique for approximating high-dimensional problems

Aiming to address the issue of the ‘curse of dimensionality’ encountered in approximating high-dimensional problems using surrogate-based methods, high-dimensional model representation (HDMR), decomposing the high-dimensional problem into summands of different-order component functions, has been widely studied. To reduce the computational demands of the current HDMR metamodelling techniques, an improved high-dimensional model representation framework (iHDMR) is presented, taking full advantage of the relationships between the first-order and second-order component functions in the Cut-HDMR theory. then, a novel metamodelling technique, termed kriging (KRG)-iHDMR, is implemented by integrating the kriging technique and the suggested surface-axes alternating sampling strategy into the iHDMR framework. The accuracy and efficiency of KRG-iHDMR are demonstrated by various numerical and engineering examples with different dimensions. Finally, an optimization problem of the multi-stage solid launch vehicle propulsion system is introduced to verify the engineering feasibility of the KRG-iHDMR metamodelling technique.

Hypothesis testing for varying coefficient models in tail index regression

AbstractThis study examines the varying coefficient model in tail index regression. The varying coefficient model is an efficient semiparametric model that avoids the curse of dimensionality when including large covariates in the model. In fact, the varying coefficient model is useful in mean, quantile, and other regressions. The tail index regression is not an exception. However, the varying coefficient model is flexible, but leaner and simpler models are preferred for applications. Therefore, it is important to evaluate whether the estimated coefficient function varies significantly with covariates. If the effect of the non-linearity of the model is weak, the varying coefficient structure is reduced to a simpler model, such as a constant or zero. Accordingly, the hypothesis test for model assessment in the varying coefficient model has been discussed in mean and quantile regression. However, there are no results in tail index regression. In this study, we investigate the asymptotic properties of an estimator and provide a hypothesis testing method for varying coefficient models for tail index regression.

High response diversity and conspecific density-dependence, not species interactions, drive dynamics of coral reef fish communities.

Species-to-species and species-to-environment interactions are key drivers of community dynamics. Disentangling these drivers in species-rich assemblages is challenging due to the high number of potentially interacting species (the 'curse of dimensionality'). We develop a process-based model that quantifies how intraspecific and interspecific interactions, and species' covarying responses to environmental fluctuations, jointly drive community dynamics. We fit the model to reef fish abundance time series from 41 reefs of Australia's Great Barrier Reef. We found that fluctuating relative abundances are driven by species' heterogenous responses to environmental fluctuations, whereas interspecific interactions are negligible. Species differences in long-term average abundances are driven by interspecific variation in the magnitudes of both conspecific density-dependence and density-independent growth rates. This study introduces a novel approach to overcoming the curse of dimensionality, which reveals highly individualistic dynamics in coral reef fish communities that imply a high level of niche structure.